Skip to content

[TRTLLM-17005][refactor] migrate attention workspace tests to a host sizing component - #22457

Open
Funatiq wants to merge 7 commits into
NVIDIA:mainfrom
Funatiq:dev/refactor/component-local-tests-7
Open

Funatiq wants to merge 7 commits into
NVIDIA:mainfrom
Funatiq:dev/refactor/component-local-tests-7

Conversation

@Funatiq

@Funatiq Funatiq commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Extract padded unfused-context workspace sizing into the PIC C++20 static tllm::common_attention_workspace library. Keep geometry and element size in AttentionUnfusedContextWorkspaceParams and retain noexcept sizing. Size Q/K/V, score buffers, and padding metadata from padded query/KV counts, while sizing token metadata from the packed count.

Apply the production integration to MHA::contextWorkspaceLayout in thop: the target branch replaced the original common/attentionOp.cpp. Preserve the replacement architecture and its fused, FP8, rotary, and Sage sizing. Keep the empty-input early return and explicitly convert its head count to the sizing interface type.

Replace attentionWorkspace.h CUDA helper dependencies with CUDA type headers and move the unchanged CUBLAS_WORKSPACE_SIZE macro into workspace.h. Link the host tests to the sizing component and retain whole-archive product linkage. Preserve the target branch layout/materialization tests and add ragged self/cross-attention, empty-input, and grouped-query cases.

Dev Engineer Review

  • Adds AttentionUnfusedContextWorkspaceParams and AttentionWorkspaceManager::buildUnfusedContextSizes(...) noexcept in attentionWorkspace.h. The sizing logic moves to attentionWorkspace.cpp, built as static library tllm::common_attention_workspace.
  • The new target requires C++20, uses PIC, and links tllm::common_headers publicly. It joins TLLM_FACADE_COMPONENT_TARGETS. The diff does not change common_src link dependencies.
  • buildUnfusedContextSizes has no zero-token early return. The diff does not show one. Empty-input handling is not established in this function.
  • MHA::contextWorkspaceLayout now calls the new function when context FMHA is off. The inline unfused mask, Q/K/V, QK, QKV, float QK, and padding-offset sizes are removed. Context FMHA uses empty unfused sizes. Separate-Q/KV FMHA sets qBuf explicitly. Fused, FP8, rotary, Sage, and multi-CTA sizing stay in place.
  • CUBLAS_WORKSPACE_SIZE (33554432) moves from cudaUtils.h to workspace.h. cudaUtils.h now includes workspace.h, so existing includers still see the macro.
  • attentionWorkspace.h drops the cudaUtils.h include and adds cuda_fp8.h and vector_types.h. Translation units that relied on the removed transitive includes may fail to compile.
  • Author-reported build and test results are not verified here. The reported counts (eight or nine CTest cases) are inconsistent across sources.

QA Engineer Review

Test files changed: cpp/tests/unit_tests/common/attentionWorkspaceTest.cpp (three new tests) and cpp/tests/unit_tests/common/CMakeLists.txt (link target and labels changed to unit;common;attention;host).

  • RaggedUnfusedSelfAttentionUsesPaddedTokenCounts: ragged self-attention with per-field literal expectations.
  • RaggedUnfusedCrossAttentionUsesPaddedTokenCounts: cross-attention, including encoderPaddingOffset from the cross extent.
  • UnfusedGroupedQuerySizesDistinguishQueryAndKvHeads: grouped-query sizes. Packed count 7 and 15 leave padded buffers and paddingOffset unchanged, and only tokensInfo scales.
  • Not covered: a zero packed-token case, MHA::contextWorkspaceLayout integration, and CI test-db or qa list entries (not established from the diff).

Coverage verdict: needs follow-up. The helper is tested directly. The production caller is not.

Per-File QA Perspective

  • cudaUtils.h: macro removed here. Verify builds still resolve it through workspace.h.
  • workspace.h: adds CUBLAS_WORKSPACE_SIZE. Verify the cuBLAS workspace stays at 32 MiB.
  • cpp/tensorrt_llm/CMakeLists.txt: adds the new target to facade components. Verify whole-archive product linking.
  • cpp/tensorrt_llm/common/CMakeLists.txt: adds the static C++20 library. Verify clean builds.
  • attentionWorkspace.cpp and .h: new unfused sizing. Verify totals match the previous inline values for ragged, cross-attention, and grouped-query inputs.
  • attentionOp.cpp: contextWorkspaceLayout uses the shared path. Verify unfused, fused, separate-Q/KV FMHA, FP8, and Sage workspace sizes.
  • cpp/tests/unit_tests/common/CMakeLists.txt: changes link and labels. Verify the test runs under unit;common;attention;host.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@Funatiq

Funatiq commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85455 [ run ] triggered by Bot. Commit: 3c7fad7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85455 [ run ] completed with state SUCCESS. Commit: 3c7fad7
/LLM/main/L0_MergeRequest_PR pipeline #70729 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Funatiq

Funatiq commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85801 [ run ] triggered by Bot. Commit: 3c7fad7 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85801 [ run ] completed with state SUCCESS. Commit: 3c7fad7
/LLM/main/L0_MergeRequest_PR pipeline #71046 completed with status: 'SUCCESS'

CI Report

Link to invocation

@Funatiq
Funatiq marked this pull request as ready for review November 19, 2026 09:41
@Funatiq
Funatiq requested review from a team as code owners November 19, 2026 09:41
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The change adds a shared method to calculate unfused attention context workspace sizes. The attention layout uses the method, and unit tests cover self-attention, cross-attention, and grouped-query sizing.

Changes

Attention Workspace Sizing

Layer / File(s) Summary
Shared workspace sizing contract and implementation
cpp/tensorrt_llm/common/attentionWorkspace.h, cpp/tensorrt_llm/common/attentionWorkspace.cpp, cpp/tensorrt_llm/common/workspace.h, cpp/include/tensorrt_llm/common/cudaUtils.h, cpp/tensorrt_llm/common/CMakeLists.txt
Adds parameters and a method for calculating unfused context workspace sizes. The implementation calculates sizes from the supplied dimensions. The cuBLAS workspace macro moves to workspace.h, and CMake defines the implementation target.
Attention layout integration
cpp/tensorrt_llm/thop/attentionOp.cpp, cpp/tensorrt_llm/CMakeLists.txt
The attention layout calls the shared sizing method when context FMHA is disabled. Context FMHA starts with empty workspace sizes, with qBuf assigned for the separate-Q/KV path. The workspace target is added to the facade component list.
Workspace sizing tests
cpp/tests/unit_tests/common/attentionWorkspaceTest.cpp, cpp/tests/unit_tests/common/CMakeLists.txt
Adds tests for ragged self-attention, cross-attention, and grouped-query sizing. The test target links the workspace implementation and adds the attention label.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Refactor

Suggested reviewers: juney-nvidia, chzblych

Merge Risk: Minimal · up to f418c

This change moves unfused attention workspace sizing into a shared helper with host tests for ragged and grouped-query cases. No concrete merge-blocking risk was identified in the supplied evidence.

Security architecture risk: Minimal · up to f418c

The refactor preserves existing sizing and execution behavior. No introduced or worsened security, ownership, or failure-containment risk was identified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated production impact is confined to in-process MHA context workspace sizing. The new module API does not itself establish a network, service, tenant, or privilege boundary; the test entrypoints are consumers rather than production dispatch interfaces.

Trust Boundaries and Controls

  • inferred — Caller-supplied geometry still determines byte extents used by the existing workspace materializer. The extraction does not add a validation boundary or grant allocation authority, and the inspected comparison does not show an existing control being removed.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title follows the repository template with a JIRA ticket and type tag, and names the main change: migrating attention workspace tests to a host sizing component. The changeset adds the tllm_common…
Description check Passed The description covers the core change. It explains the extraction of unfused-context workspace sizing into a static library, the AttentionUnfusedContextWorkspaceParams struct, the thop integration in…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title follows the repository template with a JIRA ticket and type tag, and names the main change: migrating attention workspace tests to a host sizing component. The changeset adds the tllm_common_attention_workspace sizing library, the buildUnfusedContextSizes function, and host tests that link to it, so the title matches the main change. It does not mention the removal of CUBLAS_WORKSPACE_SIZE from cudaUtils.h or the thop integration, which is expected for a title of this length.

Full details: Description check

Explanation

The description covers the core change. It explains the extraction of unfused-context workspace sizing into a static library, the AttentionUnfusedContextWorkspaceParams struct, the thop integration in MHA::contextWorkspaceLayout, the header dependency cleanup, the CUBLAS_WORKSPACE_SIZE move, the linkage changes, and the added test cases. The author checked the PR checklist. The 'Description' and 'Test Coverage' sections are present, but the Test Coverage section still contains only the template comment and does not list the tests. The description also has no explicit ticket/type line, but the title carries it. Since the description is mostly complete, it passes, with a suggestion to fill in Test Coverage.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 3c7fad7 and f418c79.

Files selected for processing: 5

  • cpp/tensorrt_llm/common/CMakeLists.txt
  • cpp/tensorrt_llm/common/attentionWorkspace.cpp
  • cpp/tensorrt_llm/common/attentionWorkspace.h
  • cpp/tensorrt_llm/thop/attentionOp.cpp
  • cpp/tests/unit_tests/common/attentionWorkspaceTest.cpp

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@yuxianq

yuxianq commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

@pengbowang-nv, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~

Assignment explanation
Componentsruntime/metadata-workspaces (65%); native-ops/attention (20%); integration/attention-tests (15%)
Tagsalgorithm:dense (100%); representation:mha (50%); representation:gqa (50%); phase:prefill (100%); features:cross-attention (100%)
PR effort5 points — Requires checking padded versus packed geometry, query/KV head counts, empty-input handling and fused/unfused caller parity across the extracted interface. This is substantial multi-file allocation work without a new attention algorithm or kernel implementation.
ImportanceImportant — Changes production attention allocation sizing, a sensitive correctness contract. In attentionOp.cpp, encoderPaddingOffset previously used batch_size * cross_kv_length; the new helper uses batchSize * kvSequenceLength, changing self-attention allocation requirements.
Retained GitHub review requestsQiJune: preserve other-group coverage: NVIDIA/trt-llm-runtime-devs, NVIDIA/trt-llm-devs
Affinity (0–1)0.845 component/tag match; 0.634 after PIC preference
Affinity evidenceMerged authorship 2 × √points; first review 1 × √points; global normalization 0.643841
Outstanding points at selection43 → 48 (+5)
Points already reserved on this PR0
Effective weight1
Completed review points (preceding 168 hours)31
Combined points after assignment79
Demand-derived target65.00 points (soft share)
Additional fair share before assignment0.00 points
Share explanationWhole PR / qualification or coverage constraint exceeds a soft share; assignment is allowed.
Selection ruleaffinity with 0.3-share guard
Assignment scope (pengbowang-nv)all

Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once.

@yuxianq
yuxianq requested review from pengbowang-nv and removed request for heyuhhh, lfr-0531 and schetlur-nv November 19, 2026 09:55

@mikeiovine mikeiovine left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stamp on behalf of runtime devs, delegating proper review to @NVIDIA/trt-llm-torch-attention-devs

@trtllm-agent-review trtllm-agent-review left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. It moves the unfused context-attention workspace sizing out of MHA::contextWorkspaceLayout into a new host-only component, tllm::common_attention_workspace, so the workspace gtest no longer needs the product library.

Reviewed at commit 3c7fad70d0 (9 files, +211/−41).

The overlap with #21915 was checked against that PR's head c64c037.

Approve: no Blocker found. 6 suggestions below are nice to have.

Nice to have (6): 6 inline, 0 below
  • (N1: Architecture and design) AttentionUnfusedContextWorkspaceParams takes numHeads and numAttnHeads, which the only caller always passes equal; different values would size the Q and score buffers from different head counts unchecked (attentionWorkspace.h:83)
  • (N1: Architecture and design) A new static library, facade entry and PUBLIC common_src link for one 30-line function whose siblings are inline in attentionWorkspace.h; no common_src source uses it (CMakeLists.txt:95)
  • (N3: Missing or ineffective tests) The ragged self/cross tests copy the helper's formulas and compare 256-byte-aligned totals, so packed-count regressions in the mask, score and padding sizes still pass (attentionWorkspaceTest.cpp:65)
  • (R1: PR process) Open PR #21915 deletes this hunk and re-implements the unfused sizing inline in UnfusedFmha::contextWorkspaceLayout; the two PRs need an agreed merge order (attentionOp.cpp:806)
  • (N4: Performance optimization opportunity) Unfused self-attention now reserves an encoderPaddingOffset slice (4 x batch x seq bytes) that the old code sized at 0 and only cross attention reads (attentionWorkspace.cpp:50)
  • (N5: Code tidiness) The zero-token early return is unreachable from production and returns a zero cuBLAS slice that the caller's layout does not match; the padded-size rationale comment was dropped (attentionWorkspace.cpp:28)
What this review checked
  • Every unfused slice except encoderPaddingOffset (attention mask, Q/K/V, QK, QKV, fp32 scores, padding offsets, tokens info, cu-seqlens) matches the deleted inline formulas. The fused path keeps its 32 MiB cuBLAS slice and its separate-Q/KV, FP8, Sage and multi-CTA sizes unchanged.
  • enqueueContext carves the workspace with the same contextWorkspaceLayout(p) it is sized with (attentionOp.cpp:1020, 1034), and zero-token input never reaches it, so the new early return cannot leave a running context call without a cuBLAS slice.
  • CUBLAS_WORKSPACE_SIZE keeps one definition and its value (workspace.h:22), and all 14 uses still reach it through cudaUtils.h. attentionWorkspaceTest runs pre-merge in test_unit_tests[common-90] on H100, and the unfused path runs pre-merge in the T5 end-to-end tests on B200.
  • Not covered: No GPU run and no full CMake build; the helper and a mutation harness were compiled on the host with stub CUDA headers.
  • Not covered: The T5 and other unfused-path end-to-end tests were not rerun; the PR's blossom-ci L0 run on 3c7fad7 passed.

Automated review by trtllm-agent-review. If a point is wrong, reply in its thread. After you push fixes or reply, the bot re-checks the Blockers within a few hours.

size_t querySequenceLength{};
size_t kvSequenceLength{};
size_t packedTokenCount{};
int numHeads{};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) numHeads sizes qkBuf/qkFloatBuf (attentionWorkspace.cpp:38) and numAttnHeads sizes qBuf/qkvBuf (attentionWorkspace.cpp:36), but nothing says how they differ. The only caller passes .numHeads = static_cast<int>(num_heads) and .numAttnHeads = mNumAttnHeads (attentionOp.cpp:811-812), and TrtllmAttention sets mNumAttnHeads = static_cast<int>(num_heads) because neither TP nor CP applies to this op (attentionVariant.h:54-55). Every new test passes equal values too. A caller that passes different values gets the Q and score buffers sized from different head counts, with no check. Could this be one numQueryHeads next to numAttnKvHeads? If two counts are really needed, please document each field with //!< and say which buffers use which.

@Funatiq Funatiq Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactor: use one query head count for attention workspace sizing

target_compile_definitions(tllm_common_cublas PUBLIC ENABLE_CUBLASLT_FP4_GEMM)
endif()

add_library(tllm_common_attention_workspace STATIC attentionWorkspace.cpp)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) buildUnfusedContextSizes is the only out-of-line member of AttentionWorkspaceManager: buildContextLayout, materializeContext and the other builders are inline in attentionWorkspace.h (244-395). Here it gets a new STATIC archive, a TLLM_FACADE_COMPONENT_TARGETS entry and a PUBLIC link from common_src (lines 126-128). None of common_src's sources include attentionWorkspace.h, and the facade entry already whole-archives the library into libtensorrt_llm. Because ${SHARED_TARGET} links common_src PUBLIC, the archive also lands on the link line of every libtensorrt_llm consumer. It also links tllm::common_error only to get its include directories; the helper uses none of that library's symbols. Either define the function inline next to its siblings and link the test to a header-only target, or keep the component and drop the common_src edge, depending on tllm::common_headers plus the CUDA include directories instead.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Narrowed attention workspace component dependencies.

constexpr int32_t kPackedTokenCount = 14;
constexpr int32_t kHeadSize = 8;

size_t expectedUnfusedContextWorkspace(bool crossAttention)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N3: Missing or ineffective tests) expectedUnfusedContextWorkspace restates the helper's formulas line by line, and both ragged tests compare only buildContextLayout(...).totalSize, where every slice is rounded up to 256 bytes. At batch 2, length 11 and 14 packed tokens with all head counts 1, a packed-count mistake often lands in the same bucket: attentionMask is 484 B padded and 308 B packed, and both round to 512. I built the helper on the host and tried nine mutations. RaggedUnfusedSelfAttentionUsesPaddedTokenCounts still passes with attentionMask, qkBuf, paddingOffset or encoderPaddingOffset sized by the packed count, and with the scores sized by the KV head count. UnfusedGroupedQuerySizesDistinguishQueryAndKvHeads catches all nine, including every one the two ragged tests catch. Suggest dropping the two ragged tests, or giving them per-field literal expectations at sizes well above 256 B (for example batch 4, length 64, 100 packed tokens) with distinct Q and KV head counts.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test: check ragged attention workspace buffer sizes directly

// smaller than batch_size * max(context q length), so sizing by max_num_tokens underestimates.
size_t const attention_mask_size = mEnableContextFMHA ? 0 : size * batch_size * input_seq_length * kv_seq_length;
auto workspaceSizes = mEnableContextFMHA ? AttentionContextWorkspaceSizes{}
: AttentionWorkspaceManager::buildUnfusedContextSizes({.elementSize = size,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) This hunk overlaps open PR #21915 ("Phased FMHA libraries, selected in Python"). At its head c64c037, MHA::contextWorkspaceSizes (thop/attentionOp.cpp:320) sizes only the fused buffers. A new UnfusedFmha::contextWorkspaceLayout (thop/phasedFmha.cpp:2525-2557) re-implements the attention mask, Q/K/V, score and padding-offset sizes inline, with the old "must upper-bound the carve" comment. If #21915 lands first, this change has to be redone there. If this PR lands first and #21915 keeps its inline copy when it rebases, buildUnfusedContextSizes has no production caller left, and the new host tests check sizing that production no longer uses. Could you link the two PRs and agree the order with the #21915 author, ideally so that UnfusedFmha::contextWorkspaceLayout calls this helper?

sizes.qkvBuf = sizes.qBuf;
sizes.qkFloatBuf = sizeof(float) * scoreElements;
sizes.paddingOffset = sizeof(int) * paddedQueryTokens;
sizes.encoderPaddingOffset = sizeof(int) * paddedKvTokens;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N4: Performance optimization opportunity) For self-attention, kvSequenceLength is input_seq_length (attentionOp.cpp:804), so this now reserves 4 * batch * input_seq_length bytes. The code it replaces used sizeof(int) * batch_size * cross_kv_length, and crossKvLength() is 0 unless the op is cross attention (attentionFeature.h:668-675). The slice is only read through cross.active() ? workspaceViews.encoderPaddingOffset : nullptr (attentionOp.cpp:1054) and in the cross-only branch (attentionOp.cpp:1435). So every unfused self-attention context call reserves workspace nothing reads; that covers T5's relative-position attention, fp32, and head sizes without an FMHA kernel. RaggedUnfusedSelfAttentionUsesPaddedTokenCounts expects the larger total, so restoring the old size fails it. Suggest passing the cross KV length in the params and sizing encoderPaddingOffset = sizeof(int) * batchSize * crossKvLength (0 for self-attention), then updating the expectation.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix: reserve encoder padding workspace only for cross attention

AttentionUnfusedContextWorkspaceParams const& params) noexcept
{
AttentionContextWorkspaceSizes sizes{};
if (params.packedTokenCount == 0)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N5: Code tidiness) Two small points on the helper:

  1. This branch has no production trigger. getWorkspaceSizeForContext returns 0 for num_tokens == 0 before it builds the layout (attentionOp.cpp:884), and the context call only runs when there are context tokens. The branch also returns cublasWorkspace = 0, while the caller then re-fills cuQSeqlens, cuKvSeqlens, cuMaskRows, rotaryInvFreq and tokensInfo, so its "empty input needs no workspace" contract is not what MHA::contextWorkspaceLayout builds. Consider dropping it together with EmptyUnfusedContextNeedsNoWorkspace, or asserting packedTokenCount > 0.
  2. The removed comment explained why these buffers use padded counts: with padding removal, the packed token count can be smaller than batch_size * max(context q length). The new doc comment (attentionWorkspace.h:240) only says "padded". One sentence there would keep the reason next to the formulas.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • fix: remove unused empty attention workspace sizing path
  • docs: explain padded attention workspace sizing

Extract padded unfused-context workspace sizing into the PIC C++20
static tllm::common_attention_workspace library. Keep geometry and element
size in AttentionUnfusedContextWorkspaceParams and retain noexcept sizing.
Size Q/K/V, score buffers, and padding metadata from padded query/KV counts,
while sizing token metadata from the packed count.

Apply the production integration to MHA::contextWorkspaceLayout in thop:
the target branch replaced the original common/attentionOp.cpp. Preserve
the replacement architecture and its fused, FP8, rotary, and Sage sizing.
Keep the empty-input early return and explicitly convert its head count
to the sizing interface type.

Replace attentionWorkspace.h CUDA helper dependencies with CUDA type
headers and move the unchanged CUBLAS_WORKSPACE_SIZE macro into workspace.h.
Link the host tests to the sizing component and retain whole-archive
product linkage. Preserve the target branch layout/materialization tests
and add ragged self/cross-attention, empty-input, and grouped-query cases.

Validation:
- Build attentionWorkspaceTest in the SM80 RelWithDebInfo devcontainer.
- Pass all nine AttentionWorkspaceManagerTest cases through CTest.
- Compile thop/attentionOp.cpp with generated project flags and -Werror;
  confirm its object references buildUnfusedContextSizes.
- Verify the test executable has no CUDA runtime or product shared-library
  dependency.
- Pass repository pre-commit checks for all changed files.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Document why Q/K/V and score buffers use padded counts: padding removal can leave fewer packed tokens than batch size times the maximum context query length. Keep the allocation rationale beside the sizing helper declaration.

Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Leave empty-context handling with MHA::getWorkspaceSizeForContext, which returns zero before building a layout. Remove the sizing helper early return and its dedicated empty-input test, whose zero-workspace contract does not describe the fields subsequently populated by the production layout builder.

Validation: rebuilt attentionWorkspaceTest and passed all eight remaining cases with CTest from tests/unit_tests/common. Top-level CTest discovery is blocked by an unrelated ucxCommTest/libpg_utils.so TllmException symbol lookup error.
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Pass the cross KV length separately from the general KV sequence length into unfused context sizing. Size encoder padding offsets from the cross extent so self-attention reserves no unused encoder slice while preserving padded K/V buffer sizing and cross-attention offsets.

Update the ragged self/cross-attention layout expectations and assert their encoder slice sizes explicitly. Keep grouped-query cross-attention coverage with an explicit cross extent.

Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases; compiled the production thop/attentionOp.cpp consumer using its generated project flags.
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Replace numHeads and numAttnHeads with numQueryHeads in the unfused context sizing parameters. Use it for both Q/QKV and score buffers, preventing callers from supplying inconsistent query head geometry. Keep the KV head count separate for grouped-query attention.

Update the production MHA caller and test initializers. The production query head counts are equal because this attention op has neither tensor nor context parallelism.

Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases, including grouped-query buffer assertions; compiled thop/attentionOp.cpp with its generated project flags; passed pre-commit checks.
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Replace the duplicated sizing formulas and aligned-total comparisons with literal per-field expectations for ragged self and cross attention. Use batch four, query length 64, 100 packed tokens, eight query heads, and two KV heads to expose packed-count and head-count mistakes. Keep explicit zero encoder padding coverage for self-attention and a distinct cross KV length of 32.

Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases. Temporary host mutants confirmed that the ragged tests detect packed attention-mask, score-buffer, and padding-offset sizing, an unwanted self-attention encoder slice, and scores sized with KV heads. Production sources were unchanged during mutation checks.
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Depend directly on common_headers and CUDA toolkit include directories instead of linking common_error solely for its include paths. Remove the unused PUBLIC common_src edge so the workspace archive does not propagate to libtensorrt_llm consumers. Keep the existing private whole-archive facade entry and direct host-test linkage.

Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases. Verified generated facade linkage retains the workspace archive exactly once with whole-archive, ucxCommTest no longer links that archive, and the host test links neither common_error nor libtensorrt_llm. The workspace archive has no undefined symbols and the host executable has no CUDA runtime dependency.
Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
@Funatiq
Funatiq force-pushed the dev/refactor/component-local-tests-7 branch from 3c7fad7 to f418c79 Compare November 20, 2026 14:10
@Funatiq

Funatiq commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86393 [ run ] triggered by Bot. Commit: f418c79 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86393 [ run ] completed with state SUCCESS. Commit: f418c79
/LLM/main/L0_MergeRequest_PR pipeline #71590 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@Funatiq

Funatiq commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86523 [ run ] triggered by Bot. Commit: f418c79 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86523 [ run ] completed with state SUCCESS. Commit: f418c79
/LLM/main/L0_MergeRequest_PR pipeline #71709 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants